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SUMMARY 

V 

This  note  is  closely  based  on  a  presentation  I  made  to  the 
SERC  Research  Area  Review  Meeting  at  the  Rutherford  Appleton 
Laboratory  on  14  December  1982.  It  is  concerned  with  the 
military  interest  in  image  processing,  and  includes  my  view  of 
where  image  processing  research  should  be  directed  in  the  future 
It  was  presented  against  a  background  of  discussion  on  how 
Pattern  Recognition  and  Image  Processing  (PRIP)  research  should 
be  co-ordinated  in  the  future,  and  whether  there  is  a  case  to 
build  up  a  number  of  'Centres  of  Excellence'  with  the  resources 
and  multidisiplinary  teams  needed  to  make  progress  in  this  field 
Since  much  of  its  content  has  a  wider  interest,  it  has  been 
published  as  an  RSRE  Memorandum. 
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1 .  INTRODUCTION 


The  presentation  was  divided  into  three  parts:  an  introduction  to  the 
military  interest  in  PRIP;  some  examples  of  military  systms  which  will 
make  significant  use  of  PRIP;  and  a  discussion  of  what  form  image 
processing  systems  might  look  like,  including  what  'tools'  they  might 
need. 

The  words  "Pattern  Recognition  and  Image  Processing"  are  often  used 
indiscriminately  to  describe  widely  differing  work,  from  image 
enhancement,  which  re-arranges  information  to  be  more  easily  interpreted 
by  human  observers,  to  earth  resource  studies  which  are  trying  to,  say, 
define  axes  in  a  multi-dimensional  image  which  correspond  to  diseased 
wheat,  to  scene  understanding  systems  which  can  interpret  various  pieces 
of  information  and  automatically  initiate  a  response.  These  activities 
are  very  different  in  the  type  of  techniques  they  use.  Image  enhancement 
has  more  to  do  with  the  limitations  of  displays  and  the  human  visual 
system,  resource  analysis  is  mainly  concerned  with  linear  transforms  and 
simple  classifiers  which  draw  class  boundaries  in  a  hyper-space,  while 
image  understanding  is  mainly  concerned  with  representing  the  constraints 
which  operate  in  the  real  world  and  applying  these  in  a  manageable 
computational  environment.  One  cannot  expect  any  sense  to  result  from 
lumping  all  these  fields  together,  and  at  the  moment  ignorance  of  these 
differences  is  one  of  the  biggest  stumbling  blocks  in  agreeing  a  policy  on 
future  PRIP  research. 

In  the  rest  of  this  paper  I  am  concerned  exclusively  with  techniques  which 
fall  into  the  image  understanding  category  (Figure  1),  particularly  those 
which  might  be  applied  in  the  late  80' s,  which  will  be  most  influenced  by 
decisions  made  today.  Figure  2  further  defines  the  scope.  In  particular 
I  am  not  concerned  with  problems  which  can  be  solved  using  decision 
theoretic  methods  only.  These  methods  are  widely  used  and  their 
performance  and  limitations  are  well  understood.  They  figure  in  many 
current  military  research  programmes,  and  are  a  standard  tool  accessible 
to  non-specialists.  There  are  standard  libraries  of  statistical 
algorithms  (eg  ALPARS);  and  military  and  commercial  exploitation  (for 
example  in  OCR  and  target  recognition)  i6  successful  and  not  in  need  of 
significant  intervention  by  SERC. 

I  make  the  assumption  that  progress  in  PRIP  will  need  advances  in  our 
fundamental  understanding  of  processing  systems,  and  not  merely  new  (and 
probably  more  powerful)  implementations  of  existing  concepts.  I 
emphasize  the  importance  of  advancing  the  state  of  the  art  in  PRIP 
techniques;  and  how  the  philosophies  of  image  understanding  can  advance 
hand  in  hand  with  developments  in  hardware,  computer  architectures,  and 
firmware. 


2.  MILITARY  INTEREST  IN  IMAGE  UNDERSTANDING 


Figure  3  states  the  two  main  reasons  why  image  processing  will  play  an 
Increasingly  important  part  in  the  future  of  Military  systems:  autonomous 
operation  and  the  need  to  make  greater  use  of  the  outputs  from  sensors. 

For  a  multitude  of  reasons,  fast  reaction  is  becoming  more  Important  in 


defence  systems.  This  forces  more  of  the  decisions  to  be  made  without 
operator  Interaction:  in  many  cases  operator  interaction  will  not  be 
possible  at  all.  Targetting,  Command  and  Control  need  to  make  sense  out 
of  various  disparate  and  possibly  un-reliable  sources,  and  will  draw  on 
the  artificial  intelligence  techniques  which  will  result  from  advanced 
image  understanding  research.  These  will  pass  control  to  autonomous 
systems  which  may  have  to  select  a  target  from  rough  instructions. 

The  trend  in  sensors  is  to  produce  more  information  than  a  man  can 
assimilate,  especially  in  a  saturation  attack  or  where  he  has  other  jobs 
to  do,  such  as  flying  a  helicopter.  As  equipment  becomes  more  complicated 
it  must  take  more  responsibility  for  its  own  adjustment  and  operation.  It 
must  still  be  effective  with  a  tired  or  undertrained  operator.  With  few 
exceptions,  we  cannot  expect  much  improvement  in  the  performance  of 
sensors  in  the  future.  Indeed,  with  the  counter-measures  likely  by  the 
90' s  they  may  be  a  good  deal  less  effective  than  they  are  today.  The  only 
available  avenue  for  advance  is  to  make  greater  use  of  the  information 
being  gathered  and  provide  sensors  with  a  greater  intelligence. 

In  imaging  systems  the  display  is  often  a  weak  link  which  can  be 
significantly  improved  by  auto-cueing  and  option  selection.  Transferring 
pictures  from  place  to  place  is  an  expensive  activity,  and  there  will  be 
considerable  pressure  to  send  coded  information  which  will  drastically 
reduce  the  bandwidths  required.  For  example,  sending  a  picture  of  a  car 
needs  the  best  part  of  a  Mbyte;  the  words  "Ford  Cortina"  use  just  a 
handful.  These  and  many  other  examples  illustrate  the  assertion  that  image 
processing  is  likely  to  be  one  of  the  most  important  techniques  in  the 
80’ s  and  90' s  for  improving  the  cost  effectiveness  of  defence  systems. 

Lack  of  capability  in  this  area  will  be  very  detrimental  to  the  ability  of 
military  (and  commercial)  products  to  meet  competitively  the  future 
requirements . 

Figure  4  shows  a  typical  military  image  processing  situation.  The  output 
from  the  sensor  may  need  to  be  manipulated  and  transformed  to  correct  for 
defects  and  to  put  the  output  into  a  suitable  processing  space.  This 
corrected  image  is  interpreted  and  then  compared  with  ground  rules,  a- 
priori  information,  some  of  which  will  be  scenario  dependent.  This 
decision  is  then  acted  upon,  pehaps  after  approval  by  a  'man  in  the  loop' . 

Figures  5  and  6  give  three  examples  of  image  processing:  a  panoramic  IR 
surveillance  sensor,  and  autonomous  missile,  and  an  advanced  radar.  The 
latter  also  illustrates  the  relationship  between  mathematical  methods, 
which  tend  to  break  down  because  the  problem  may  be  ill-formed,  and 
intelligent  guesses  based  on  a-priori  information  which  might  have  a  good 
chance  of  getting  the  correct  solution  even  when  an  unconstrained  purely 
mathematical  approach  would  'blow  up'.  An  intelligent  guess  is  better 
than  no  deduction  at  all.  Many  might  not  consider  this  PRIP  in  a  true 
sense,  but  I  would  not  agree.  It  illustrates  the  generality  image 
understanding  methods  are  likely  to  have. 


3.  WHAT  DIRECTION  SHOULD  IMAGE  UNDERSTANDING  RESEARCH  BE  TAKING? 


3.1  Introduction  to  the  Problems  Image  Understanding  Must  Solve 

Let  us  say  that  there  are  two  types  of  image  understanding  problem 
(Figure  7):  simple  problems  where  the  dimensionality  is  small  and  there  is 
a  good  training  regime  available;  and  more  a  general  case  where  the 
dimensionality  is  large  and  training  is  poor,  ie  it  i6  difficult  or 
impossible  to  gather  a  large  enough  training  set  to  establish  decision 
boundaries  with  adequate  precision.  In  the  first  simpler  case,  decision 
theoretic  approaches  will  give  an  'optimum'  classification,  at  least  when 
Gaussian  statistics  apply.  As  the  dimensionality  and  number  of  classes 
increase  the  size  of  the  training  set  grows  explosively  to  the  point  where 
decision  theoretic  methods  cannot  work  even  given  enormous  comyytlng 
power;  it  is  clearly  unsanitary  to  gather  a  training  set  of  10  or  more 
independent  measurements. 

What  is  needed  to  get  around  this  problem  is  a 'way  of  constraining  the 
problem  sufficiently  to  get  a  solution.  How  this  is  best  done  is  one  of 
the  great  challenges  facing  science  at  the  moment.  It  can  be  done  by 
reducing  the  dimensionality,  which  implies  some  non-linear  operation  such 
as  coding,  eliminating  unlikely  or  impossible  solutions,  for  example  by 
incorporating  Newton's  Laws  or  the  rules  that  relate  3D  objects  to  2D 
views.  It  can  also  be  done  by  model  generation  and  testing  where 
inductive  methods  dominate,  or  by  special  grammars  which  contain  the 
constraints  of  the  real  world.  It  is  most  likely  that  this  can  only  be 
solved  by  systems  which  have  a  considerable  degree  of  internal  structure: 
particular  parts  of  the  process  will  perform  different  specialized 
operations. 

The  progress  in  computing  hardware  and,  of  equal  importance,  computer 
architectures,  will  provide  a  significant  increase  in  computing  power 
over  the  next  decade.  You  can  argue  about  what  gain  will  be  achieved  in 
terms  of  chip  or  wafer  complexity,  and  what  increases  in  efficiency  and 
real  estate  utilization  will  come  from  architecture  advances,  but  an 
increase  in  the  100-1000  region  is  not  impossible.  By  the  mid  90's  the 
amount  of  computing  power  available  to  an  image  processing  system  will  be 
so  much  greater  that  new  approaches  to  the  way  image  processing  is 
per  jrmed  will  be  necessary.  We  will  need  a  new  generation  of  researchers 
who  have  a  good  grasp  of  computer  architectures,  philosophy  and 
mathematics,  working  in  teams  which  comprise  experts  in  each  of  these. 

As  the  complexity  of  each  'chip'  tends  towards  wafer  size,  the  penalty  of 
using  a  limited  number  of  standard  components  will  be  severely  felt,  even 
if  these  are  relatively  general  purpose,  and  the  main  advances  are  most 
likely  to  be  made  by  teams  with  access  to  powerful  CAD  facilities  who  can 
fabricate  large  custom  chips  from  well  tried  sub-modules.  This  approach 
will  become  the  more  important  as  getting  data  on  and  off  the  substrate 
and  matching  real  to  virtual  machines  assume  a  greater  priority  in  the 
design.  At  the  same  time  computer  architecture  will  play  a  larger  part  in 
the  concepts  of  image  understanding,  and  the  arguments  for  close 
association  of  algorithm,  paradigm,  computer,  and  chip  development  will 
strengthen.  Image  processing  is  very  likely  to  be  the  biggest  and  most 
demanding  user  of  VLSI  in  the  future;  the  IP  community  has  a 
responsibility  to  steer  future  chip  development  (Figure  8). 


3.2  An  Approach  to  Implementing  Image  Understandin, 

A  theme  which  is  explored  by  this  paper  is  to  define  relatively 
independent  "Functional  Blocks"  which  can  be  configured  into  a  complete 
system.  Research  can  then  be  aimed  at  enhancing  the  performance  of  these 
Functional  Blocks,  which  can  be  transportable  either  through  the  use  of 
one  from  a  number  of  suitable  languages  (eg  Fortran,  Pascal  and  Prolog) 
through  a  standard  library,  or  eventually  as  ROM'S  for  commonly  used 
machines  or  even  as  VLSI  sub-masks  in  a  chip  CAD  simulator.  When 
considering  this  concept  one  must  avoid  being  constrained  by  past 
difficulties  in  standardization.  Software  can  be  embodied  into  system 
generation  procedures  in  very  flexible  ways,  and  technology  has  advanced 
to  the  point  where  hardware  is  designed  using  methods  previously  confined 
to  software.  Indeed  the  distinction  between  hardware  and  software  is 
becoming  increasingly  academic,  opening  very  powerful  ways  of 
Incorporating  systematic  knowledge  into  CAD  facilities. 

Figure  9  shows  how  typical  Functional  Blocks  might  be  arranged  to  form  an 
image  understanding  system.  Following  attention  direction,  a  portion  of 
the  scene  is  coded  into  a  feature  space  by  a  variety  of  operators.  These 
work  over  very  local  regions;  they  might  use  statistical  feature 
classifiers,  rule  based  or  heuristic  methods.  The  feature  map  represents 
a  highly  non-linear  transform  space  with  dimensions  which  represent,  say, 
the  "edginess"  or  "corneriness"  of  regions  in  the  attention  patch.  In  the 
classic  Marr  tradition,  shape  operators  then  scan  this  feature  map  and 
find  more  global  properties  such  as  rectangles,  arcs,  sticks,  conic 
sections,  etc.  This  process  will  necessarily  have  to  make  a  number  of 
alternative  interpretations  because  of  the  ambiguities  in  the  way  features 
can  be  connected.  To  resolve  these  ambiguities  these  shapes  are  used  as 
keywords  to  trigger  an  expert  system  which  would  contain  a  data  base  of 
possible  objects.  It  can  then  ask  questions  of  the  feature  map,  or  even 
of  the  original  scene,  until  it  has  enough  information  to  make  a  decision. 
This  can  be  looked  upon  as  Interactively  building  a  model  of  the  scene 
U6lng  the  ground  rules  contained  in  the  expert  system,  and  testing  it 
against  what  is  observed.  The  feature  coding  and  the  shape  extraction 
form  a  deductive  loop  which  drives  the  guesses  made  by  the  expert  system; 
the  expert  system  and  model  testing  form  and  inductive  loop  which  tries 
out  these  guesses. 

Clearly  this  is  one  from  many  possible  ways  of  carrying  out  image 
understanding.  The  point  I  make  is  its  use  of  a  number  of  distinct  levels 
of  operations  and  functional  blocks  which  have  wider  application.  If 
effective  algorithms  existed  today  it  would  be  possible  to  try  out  such  a 
configuration,  see  how  it  worked  and  improve  its  design.  Unfortunately  it 
is  not  yet  possible  to  lay  ones  hand  on  robust  feature  coding  algorithms; 
expert  systems  are  mostly  "hand  crafted"  and  very  difficult  to  train;  and 
methods  for  representing  shapes  are  very  immature.  Relaxation  labelling 
is  on  a  fairly  firm  theoretical  footing,  but  applying  it  to  model 
generation  is  not  a  straight  forward  step.  The  aims  of  those  engaged  in 
techniques  research  should  be  to  bring  these  and  other  functional  blocks 
to  a  configurable  state,  in  an  appropriate  computer  architecture(s) . 

Figure  10  lists  some  other  Functional  Blocks  which  should  be  covered  by  a 
coherent  image  processing  research  programme. 


3.3  List  of  Research  Topics  Important  to  Image  Understanding 

It  is  useful  to  discuss  the  areas  of  PRIP  research  against  a  background  of 
broad  topics.  I  have  produced  the  following  list  of  such  topics  to 
simplify  discussion  of  image  understanding,  not  necessarily  as  a  guide  to 
how  research  should  eventually  be  coordinated. 

a.  Local  Region  Evaluation.  This  covers  methods  which  categorize  the 
properties  around  small  portions  of  a  scene  including  texture, 
movement,  shape:  ie  low  level  sketch  generators. 

b.  Propagation  Methods.  This  is  to  do  with  associating  and  relating 
different  parts  of  an  image.  It  includes  existing  concepts  of 
segmentation,  region  growing,  and  associating  features  like  edges  and 
corners  to  form  higher  level  shapes.  The  rules  of  association  will 
become  quite  complex,  (eg  would  include  2D  to  3D  interpretations)  and 
are  likely  to  draw  on  the  methods  of  topic  d. 

c.  Cellular  and  Systolic  approaches.  These  descendents  of  the 
"Perceptron"  concept  include  associative  memories,  hierarchies  of 
nodes  with  trainable  regions  of  sensitivity  in  the  planes  below  them, 
'Wizard'  machines,  etc. 

d.  Artificial  Intelligence  using  Knowledge  Based  approaches.  Already 
a  growth  area,  this  is  likely  to  be  the  core  of  any  image 
understanding  system.  Expert  systems  appear  to  be  essential  at  the 
higher  levels  of  any  PRIP  system,  but  less  obviously,  they  also  have 
potential  impact  at  lower  levels. 

e.  Linguistic  Methods.  Success  in  the  description  of  pictures  by 
linguistic  forms,  which  also  conform  to  a  grammar  which  somehow 
contain  the  constraints  of  the  real  world,  would  have  great 
significance.  In  some  respects  this  is  an  alternative  to  the  use  of 
Expert  Systems. 

f.  Virtual  Computing  Machines  and  Languages.  Much  insight  into  the 
physics  of  information  processing  has  been  gained  from  the  writing  and 
implementation  of  computer  languages.  As  well  as  providing  the  tools 
to  execute  image  understanding.  Computer  Science  must  have  a  lot  to 
6ay  about  the  fundamental  limitations  and  properties  of  information 
manipulation. 

g.  Hardware  design.  Includes  implementing  appropriate  virtual 
machines  efficiently,  providing  good  image  processing  development 
systems,  adapting  concepts  to  fit  well  in  VLSI,  and  also  interacting 
with  the  other  topics  to  ensure  optimum  use  can  be  made  of  device 
performance. 

h.  Study  of  Complete  Processing  Systems.  A  natural  part  of  the 
'functional  block'  concept,  an  important  area  is  determining  how  these 
interact  as  a  system. 

i.  Comparisons  with  Biolgical  Systems.  There  are  several  areas  of 
interest,  particularly  the  low  level  operation  of  biological  vision, 
deductions  about  its  overall  structure,  biochemical  engineering,  etc. 
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If  this  country  could  point  to  strong  teams  working  in  each  of  these 
topics,  we  would  have  the  makings  of  a  good  image  understanding  programme. 
It  is  of  concern  that  this  is  not  the  case:  out  of  245  entries  in  the 
"Pattern  Recognition  and  Image  Processing  Review"  recently  produced  by  Dr 
J  V  Kittler  of  RAL,  which  lists  all  PRIP  research  in  universities  and  SERC 
labs,  only  31  fall  into  the  topics  defined  above.  Research  in  these 
topics  must  be  strengthened  as  a  matter  of  urgency,  and  equally  important, 
made  to  inter-relate  to  provide  the  foundations  for  future  PRIP  systems. 


4.  CONCLUSIONS 


It  has  been  suggested  that  an  initiative  is  needed  to  establish  a  strong 
research  base  in  the  UK  to  make  sure  we  have  the  people  and  equipment  to 
make  the  necessary  impact.  I  stongly  agree  with  this  proposal.  It  should 
encompass  the  work  at  research  establishments  like  RSRE  and  Rutherford 
Labs,  the  universities,  and  provide  facilities  and  knowledge  to  industry. 

I  am  interested  to  see  three  things  come  about: 

a.  To  see  that  research  into  new  techniques  is  supported  in  a 
coordinated  manner  at  the  universities,  so  that  at  least  the  topics 
outlined  in  section  3.3  are  being  adequately  pursued.  When  looked  at 
as  a  whole,  the  long  term  research  should  form  a  coherent  picture. 

The  universities  must  accept  that  long  term  research  needs  to  be 
planned  around  objectives,  and  not  conducted  around  the  3  year  PhD  in 
an  organisational  vacuum.  Correctly  implemented,  it  should  free 
university  research  from  the  often  inappropriate  alternative  of 
linking  their  research  to  short  term  applied  problems  in  order  to  gain 
funding. 

b.  There  should  be  a  number  of  'Centres  of  Excellence'  with  teams 
large  enough  to  be  able  to  tackle  the  multi-rdisciplinary  nature  of 
this  research  with  the  appropriate  hardware,  which  is  too  expensive  to 
be  proliferated  throughout  universities,  MOD  and  Industry.  These 
centres  would  form  the  meeting  point  for  long  term  and  applied 
research.  It  will  be  possible  to  formulate  research  programmes  whose 
primary  aim  is  to  further  the  techniques  base,  independent  of 
applications,  but  will  be  able  to  use  the  expertise  developed  and 
maintained  by  these  programmes  on  appropriate  applied  work.  That  is, 
individuals  will  have  a  coordinated  responsibility  to  advance 
techniques,  but  will  regularly  become  involved  with  applications. 

This  method  of  operation  is  impossible  without  the  Centres  of 
Excellence.  Because  of  its  association  with  sensors,  computer 
devlce6/VLSl,  and  image  understanding  applied  to  systems,  I  argue  that 
RSRE  should  be  one  of  these  centres.  All  PRIP  researchers  in 
universities,  industry  and  government  departments  would  presumably 
spend  some  of  there  time  at  one  of  these  centres. 

c.  There  is  already  a  shortage  of  adequate  under-graduate  or  post¬ 
graduate  training  in  PRIP.  The  subject  has  so  far  not  received  the 
status  or  interest  it  deserves  In  universities,  partly  because  of  the 
Lighthill  report  which  blighted  work  during  the  70' s.  As  image 
understanding  assumes  greater  importance  there  will  be  a  severe 
shortage  of  experienced  graduates.  Satisfying  this  requirement  may  in 
some  sense  in  conflict  with  the  Centres  of  Excellence  idea,  since  it 
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tends  to  argue  for  setting  up  image  processing  departments  in  all  the 
major  universities.  It  is  a  conflict  which  must  be  correctly 
resolved,  and  not  used  as  an  argument  against  establishing  'Centres  of 
Excellence' . 

To  bring  these  about  will  not  be  easy,  especially  in  a  timely  manner.  An 
attempt  must  be  made  if  we  are  to  be  able  to  retain  any  Image 
Understanding  research  initiative  in  the  UK. 

Figure  11  lists  a  number  of  points  I  would  like  people  to  take  away  with 
them: 

Image  Processing  is  bound  to  become  a  vital  technique  in  most  future 
military  systems  (and  many  commercial  applications  as  well). 

The  UK  research  into  image  processing  techniques  is  in  a  poor  state 
and  has  fallen  behind  other  countries. 

The  multi-disciplinary  nature  of  the  subject,  the  need  to  achieve  a 
good  relationship  between  speculative  and  applied  research,  and  the 
high  cost  of  capital  equipment,  make  the  setting  up  of  'Centres  of 
Excellence'  paramount. 

More  emphasis  must  be  given  to  teaching  image  understanding  in 
universities  as  a  legitimate  multi-disciplinary  field. 

Is  is  possible  to  link  long  term  research  with  a  plan  for  providing  a 
set  functional  blocks  which  will  form  the  building  bricks  for 
applications  work? 
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IMAGE  ENHANCEMENT 
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THIS  PAPER  WILL  NOT  INCLUDE  PATTERN  RECOGNITION 
PROBLEMS  WHICH  CAN  BE  TREATED  BY  FERTURE 
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WHY  IS  THERE  MILITARY  INTEREST  IN  IP 

NEED  FOR  nUTONOMOUS  OPERRTION 
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EXAMPLES  OF  MILITARY  IMAGE  PROCESSING 
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